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DETAILED ACTION 



This communication is responsive to the RCE, filed 06/06/05. 

Claims 1-13, 15-28, and 30-35 are pending in this application, and claims 1, 6, 7, 9, 10, 
12, 17, 22, 26, 27, 28, 30, and 31 are independent claims. In this communication, all independent 
claims are amended. This action is made non-final. 

Claim Rejections - 35 USC §103 

1. The following is a quotation of 35 U.S. C 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

2. Claims 1, 4, 5-7, 9, 10, 12, 15, 16, 22, 24, 25, 26-28, and 30 are rejected under 35 U.S.C 
103(a) as being unpatentable over Kuzunuki et al (U.S. Patent No. 5,917,490). 

As to claim 1, Kuzunuki teaches an information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space (display 101, 
col. 6 lines 14-32, block 400, col. 6 lines 44-58, and fig. 1); 

a display means for displaying an image on said operation surfaces (e.g., col. 6 lines 14- 
32, projectors 105 and 107, col. 7 lines 1 1-29, and fig. 2); 
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an imaging means for picking up an image of said operation surfaces (e.g., col 2 lines 

41- 51, overhead camera 300 to pick up actual images, col. 6 lines 1-12, and figs. 1-2); 

a stationary environment type computer arranged in said information space (e.g., col. 6 
lines 1-32, lines 44-58, and figs. 1-2); and 

one or more than one physical objects mounted on said operation surfaces and containing 
a visually identifiable visual marker on the surface thereof (a barcode is assigned to an objects, 
col. 21 lines 20-26, lines 42-58); 

said environmental type computer being adapted to execute (a program which is executed 
by the information processing unit, col. 19 lines 41-65); 

a processing operation of recognizing the identification information and the position 
information of each of said physical objects by identifying the visual marker on the surface of 
each of said physical objects (e.g., the barcode is assigned to an objects, col. 21 lines 20-26, lines 

42- 58); 

a processing operation of recognizing the digital object dropped to a site on the surface of 
each of said physical objects (e.g., col. 7 lines 46-65 and col. 14 lines 6-9); and 

a processing operation of forming link information for linking the digital object to the 
dropped site on the surface for each of said physical objects (e.g., col. 3 lines 44-52, link between 
physical objects and electronically stored information, col. 19 lines 36-54); 

however, Kuzunuki does not clearly show the visual marker (bar code) defining position 
information for the physical objects. It would have been obvious to a person with ordinary skill 
in the art at the time of the invention to have a bar code or an identification on the object to ease 
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the user in recognizing and locating the object for accuracy and efficiency when organizing the 
objects. 

As to claim 4, Kuzunuki teaches the information input/output system according to claim 
1, wherein 

said imaging means can identify the position indicated by an optical pointer for indicating 
a specific position by irradiating a beam of light with a predetermined wavelength (using infrared 
light, col 21 lines 20-26); and 

the user is allowed to indicate a position in said information space by means of 
coordinates and the optical pointer (a barcode reader, col. 21 hnes 20-26). 

As to claim 5, Kuzunuki teaches the information input/output system according to claim 
1, wherein said environment type computer executes a processing operation of calling the linked 
digital object and/or displaying the digital object to the user in response to a user operation 
applied to the site of forming the link information on the surface of each of the physical objects 
on the basis of the image picked up by said imaging means (e.g., col. 3 lines 44-52 and col. 19 
lines 41-65), 

As to claims 6-7, they are similar in scope to claim 1 above; therefore, rejected under a 
similar rationale. 

As to claim 9, Kuzunuki teaches an information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space; a display 
means for displaying an image on said operation surfaces; an imaging means for picking up an 
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image of said operation surfaces; a stationary environment type computer arranged in said 
information space; and one or more than one physical objects mounted on said operation surfaces 
and containing a visually identifiable visual marker on the surface thereof (note the rejection of 
claim 1 above); 

at least one of said physical objects being a virtual camera having a virtual imaging 
direction as specified by the position information contained in the visual marker (e.g., the 
facsimile direction to be placed within a given distance, col. 2 line 66-col. 3 line 1 1). 

As to claim 10, Kuzunuki teaches the information input/output system according to claim 
9, wherein 

said environmental type computer is adapted to execute (e.g., col. 19 lines 41-65); 

a processing operation of recognizing/identifying said virtual camera and identifying the 
position information and the virtual imaging direction (a video camera 6 mounted above a desk 
surface 2 for picking up physical documents, col. 6 lines 1-7, and fig. 1); 

a processing operation of generating a virtual picked up image according to the position 
information and the virtual imaging direction (e.g., col. 2 Hne 66-col. 3 line 1 1, and col. 10 line 
56-col. 1 1 line 27); and 

a processing operation of displaying the virtual picked up image to the user (figs. 6A-8B). 

As to claim 12, Kuzunuki teaches an information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space; a display 
means for displaying an image on said operation surfaces; an imaging means for picking up an 
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image of said operation surfaces; a stationary environment type computer arranged in said 
information space; one or more than one physical objects mounted on said operation surfaces and 
containing a visually identifiable visual marker on the surface thereof (note the rejection of claim 
1 above); and 

an optical pointer adapted to point a specific position by irradiating a beam of light with a 
predetermined wavelength (using infrared light, col. 21 Unes 20-26); 

said imaging means being capable of identifying the position pointed by said optical 
pointer (a barcode reader, col. 21 lines 20-26). 

As to claim 13, Kuzunuki teaches the information input/output system according to claim 
12, wherein 

said environment image type computer executes on the basis of the picked up by said 
imaging means (e.g., col. 19 lines 41-65); 

a processing operation of recognizing the user operation on said operation surfaces, using 
said optical pointer (a barcode reader, col. 21 lines 20-26); and 

a processing operation of controlling the display of the digital object by said display 
means according to the result of the recognition (e.g., col. 21 lines 20-58). 

As to claim 15, Kuzunuki teaches an information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information pace; a display 
means for displaying an image on said operation surfaces; an imaging means for picking up an 
image of said operation surfaces; a stationary environment type computer arranged in said 
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information space; one or more than one portable computers capable of exchanging digital 
objects with other computers; one or more than one physical objects mounted on said operation 
surfaces and containing a visually identifiable visual marker on the surface thereof (note the 
rejection of claim 12 above); and 

a portable ID recognition device adapted to recognize the visual marker on the surface of 
each of said physical objects and being capable of holding or releasing the recognized 
identification information (e.g., col. 21 lines 20-58). 

As to claim 16, Kuzunuki teaches the information input/output system according to claim 
15, wherein 

said environment type computer executes (e.g., col. 19 lines 41-65); 

a processing operation of obtaining a digital object corresponding to the held 
identification information in response to the holding operation of said ID recognition device 
(e.g., col. 21 lines 20-58); and 

processing operation of transferring a digital object corresponding to the held 
identification information to a nearby physical object in response to the releasing operation of 
said ID recognition device (e.g., col. 21 lines 20-58). 

As to claim 22, it is individually similar in scope to claim 1 above; therefore, rejected 
under similar rationale. 

As to claim 24, it is individually similar in scope to claim 4 above; therefore, rejected 
under similar rationale. 

As to claim 25, Kuzunuki teaches the information input/output system according to claim 
22, further comprising: 
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a step of calling the linked digital object and/or displaying the digital object to the user in 
response to a user operation applied to the site of forming the link information on the surface of 
each of the physical objects (e.g., col. 3 lines 44-52, link between physical objects and 
electronically stored information, col. 19 lines 36-54). 

As to claim 26, it is individually similar in scope to claim 7 above; therefore, rejected 
under similar rationale. 

As to claim 27, it is individually similar in scope to claim 1 above; therefore, rejected 
under similar rationale. 

As to claim 28, it is individually similar in scope to claim 13 above; therefore, rejected 
under similar rationale. 

As to claim 30, it is individually similar in scope to claim 16 above; therefore, rejected 
under similar rationale. 

3. Claim 2, 3, 8, 11, 17-21, 23, and 31-35 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Kuzunuki et al, (U.S. Patent No. 5,917,490) in view of Pierre Wellner ("The 
Digital Desk Calculator," Proceedings of ACM Symposium on User Interface Software and 
Technology (UIST '91), 1M3 Nov. 1991). 

As to claim 2, the modified Kuzunuki teaches the information input/output system 
according to claim 1, wherein at least one of said physical objects is capable of being moved in 
said information space and exchanging digital objects with others (note the rejection of claim 1 
above); however, the modified Kuzunuki still does not teach that the physical object is a portable 
computer capable of being moved in said information space and exchanging digital objects with 
other computers. An electronic object (physical object) of Wellner clearly discloses concept of 
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controlling/interacting with the electronic object by scanning, and then projecting it back on an 
operating interface (see page 30 column 1-2), Moreover, Wellner's invention is not only limited 
that the electronic object is a DigitalDesk Calculator but also meant the same concept of using 
the electronic object as mentioned above can be applied on any regular PC if it gets scanned and 
projected on the operating interface ("the DigitalDesk Calculator is an example that illustrates 
how merging these two worlds can improve the usability of a very simple and well-established 
apphcation," sic, see Conclusion on page 32). It would have been obvious at the time of the 
invention that a person with ordinary skill in the art would want to have the system allows the 
user to interact with the electrical object of Wellner in the modified environment of Kuzunuki to 
share computer functionality to the physical objects and environment while allowing users to 
continue interacting with the objects as before (Wellner, column 1, page 32). 

As to claims 3 and 23, they are similar in scope to claim 2 above; therefore, rejected 
under similar rationale. 

As to claim 8, it is similar in scope to claim 2 above; therefore, rejected under similar 
rationale. 

As to claim 1 1, it is similar in scope to claim 2 above; therefore, rejected under similar 
rationale. 

As to claim 17, the modified Kuzunuki in view of Wellner teaches an information 
input/output system to be used for user operations relating to an object in an information space 
realized by expanding a digital space of a computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space; a display 
means for displaying an image on said operation surfaces; an imaging means for picking up an 
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image of said operation surfaces; a stationary environment type computer arranged in said 
information space; one or more than one portable computers capable of exchanging digital 
objects with other computers; and one or more than one physical objects mounted on said 
operation surfaces and containing a visually identifiable visual marker on the surface thereof 
(note the rejections of claims 2 and 12 above); 

said environment type computer being adapted to allow mouse/cursor operations on said 
portable computers to extend to said operation surfaces and display a rubber band as visual 
feedback (actual object handling, col. 9 lines 6-29) according to the area on said operation 
surfaces as defined by means of said mouse/cursor (note the rejection of claim 2 above). 

As to claim 18, it is similar in scope to claim 17 above; therefore, rejected under similar 
rationale. 

As to claim 19, Kuzunuki teaches the information input/output system according to claim 
17, wherein another object can apply a process or a method it possesses to the rubber-banded 
original digital object in response to an action of said original object of being dropped on said 
another object (actual object handling, col 9 Unes 6-29, and figs. 6A-8B). 

As to claim 20, Kuzunuki teaches the information input/output system according to claim 
19, wherein said another object is a physical object possessing a process or a method for printing 
a document (printers, col. 15 lines 1-10), 

As to claim 21, Kuzunuki teaches the information input/output system according to claim 
19, wherein said another object is a name card possessing a process or a method for mailing a 
document to the mail address corresponding to the name on it (e.g., col. 8 lines 15-31, and fig. 
3A). 
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As to claim 31, it is individually similar in scope to claim 17 above; therefore, rejected 
under a similar rationale. 

As to claim 32, it is individually similar in scope to claim 18 above; therefore, rejected 
under a similar rationale. 

As to claim 33, it is individually similar in scope to claim 19 above; therefore, rejected 
under a similar rationale. 

As to claim 34, it is individually similar in scope to claim 20 above; therefore, rejected 
under a similar rationale. 

As to claim 35, it is individually similar in scope to claim 21 above; therefore, rejected 
under a similar rationale. 

Response to Arguments 
4. Applicant's arguments with respect to claims have been considered but are moot in view 
of the new ground(s) of rejection. 



Conclusion 

5. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

Gombrich et al. (U.S. Patent No. 4,835,372) teach a bar code to define a location 
of an object (col. 18 lines 17-20). 

Sanford (U.S. Patent No. 5,334,822) teaches using an identification or bar code to 
track products (cols. 1-8 and figs. 2-13). 
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6. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to True T. Chuong whose telephone number is 571-272-4134. The 
examiner can normally be reached on M-Th and alternate Fridays 8:30 AM - 5:00 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Weilun Lo can be reached on (571) 272-4847. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an appUcation may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or PubUc PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



True T. Chuong 
08/19/05 



